Information processing apparatus, information processing method, and storage medium

ABSTRACT

An information processing apparatus includes an obtaining unit configured to obtain information of a plurality of labels applied to the learning data by a plurality of users, information regarding reliability of each applied label itself, and information regarding reliability of a user who applies the relevant label, wherein the information of the label is information regarding a result to be recognized in a case where the predetermined recognition is performed on the learning data and a determination unit configured to determine a label to the learning data from among the plurality of labels based on the reliability of the label itself and the reliability of the user who applies the relevant label.

BACKGROUND Field of the Disclosure

The present disclosure relates to a labeling technique for applying alabel to data.

Description of the Related Art

Techniques for constructing prediction models from data groups andcalculating prediction to query data pieces by supervised machinelearning are used in various field such as object recognition. Thesupervised learning accompanies an operation for labeling learning datapieces in advance to apply a label expected as an output result togetherwith query data. Generally, as the number of the labeled learning datapieces to which correct labels are applied increases, accuracy of theprediction model is improved. Thus, methods for labeling learning datapieces using a large amount of manpower and methods for automaticallyperforming labeling have been conventionally proposed.

Japanese Patent Application Laid-Open No. 2016-62544 describes a methodfor automatically performing labeling by calculating a ranking ofdistances in a plurality of characteristics among learning data pieces.

However, a method using a large amount of manpower includes variation ofeach person, and there is a possibility that a false label is applied.In addition, according to the method described in Japanese PatentApplication Laid-Open No. 2016-62544, there is no choice but to trustthe result of labeling automatically performed, and a countermeasure toa case when an error occurs in labeling is not considered.

SUMMARY

The present disclosure is directed to determining appropriately a labelwhen a false label is applied by learning data.

An information processing apparatus according to the present disclosureincludes an obtaining unit configured to obtain information of aplurality of labels applied to the learning data by a plurality ofusers, information regarding reliability of each applied label itself,and information regarding reliability of a user who applies the relevantlabel, wherein the information of the label is information regarding aresult to be recognized in a case where the predetermined recognition isperformed on the learning data and a determination unit configured todetermine a label to the learning data from among the plurality oflabels based on the reliability of the label itself and the reliabilityof the user who applies the relevant label.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofa labeling system according to one or more aspects of the presentdisclosure.

FIG. 2 illustrates an example of correspondence relationships amongpieces of data, labels, and attribute information to be managedaccording to one or more aspects of the present disclosure.

FIG. 3 illustrates examples of monotonically increasing functions f(x)according to one or more aspects of the present disclosure.

FIG. 4 illustrates examples of label determination of an areadesignation type according to one or more aspects of the presentdisclosure.

FIG. 5 illustrates an example of processing on bounding boxes accordingto one or more aspects of the present disclosure.

FIG. 6 is a flowchart illustrating processing for labeling.

FIG. 7 is a flowchart illustrating processing for determining a labelaccording to one or more aspects of the present disclosure.

FIG. 8 is a block diagram illustrating an example of a configuration ofa labeling confirmation system according to one or more aspects of thepresent disclosure.

FIG. 9 illustrates an example of display for editing a symboldesignation type label on a touch panel type display according to one ormore aspects of the present disclosure.

FIG. 10 illustrates an example of display for editing an areadesignation type label on a touch panel type display according to one ormore aspects of the present disclosure.

FIG. 11 is a flowchart illustrating a typical flow for applying andconfirming a label using the labeling confirmation system according toone or more aspects of the present disclosure.

FIG. 12 is a block diagram illustrating an example of a configuration ofa labeling system based on crowdsourcing according to one or moreaspects of the present disclosure.

FIG. 13 is a flowchart illustrating a flow of labeling based oncrowdsourcing according to one or more aspects of the presentdisclosure.

FIG. 14 is a block diagram illustrating an example of a configuration ofa learning and recognition system based on crowdsourcing according toone or more aspects of the present disclosure.

FIG. 15 is a flowchart illustrating a flow of learning and recognitionbased on crowdsourcing according to one or more aspects of the presentdisclosure.

FIG. 16 is a block diagram illustrating an example of a configuration ofa labeler evaluation system according to one or more aspects of thepresent disclosure.

FIG. 17 is a flowchart illustrating a flow for evaluating a labelerusing the labeler evaluation system according to one or more aspects ofthe present disclosure.

FIG. 18 illustrates an example of a hardware configuration of aninformation processing apparatus according to one or more aspects of thepresent disclosure.

DESCRIPTION OF THE EMBODIMENTS

Before describing each exemplary embodiment according to the presentdisclosure in detail, a hardware configuration of an informationprocessing apparatus according to each exemplary embodiment is describedwith reference to FIG. 18. In FIG. 18, a central processing unit (CPU)1810 comprehensively controls each device connected via a bus 1800. TheCPU 1810 reads out and executes a processing step and a program storedin a read-only memory (ROM) 1820. Each processing program, devicedriver, and the like including an operating system (OS) according to thepresent exemplary embodiment which are stored in the ROM 1820 aretemporarily stored in a random access memory (RAM) 1830 and executed bythe CPU 1810 appropriately. An input interface (I/F) 1840 inputs aninput signal in a format which can be processed by the informationprocessing apparatus from an external apparatus (an image capturingapparatus or the like). An output I/F 1850 outputs a processing resultby the information processing apparatus according to the presentdisclosure as an output signal to the external apparatus in a formatwhich can be processed by the external apparatus.

An information processing apparatus 10000 according to a first exemplaryembodiment stores a result that a person or an algorithm (i.e., alabeler) labels each learning data in association with reliability ofthe labeler and a confidence degree of labeling by each labeler asattribute information pieces. Further, the information processingapparatus 10000 determines a likely label based on a label applied to atarget learning image and a reliability of the label. A person or analgorithm performing labeling is referred to as a labeler appropriately.

FIG. 1 is a block diagram illustrating an example of a configuration ofan information processing apparatus (a labeling apparatus) 10000according to the first exemplary embodiment.

A learning data storage unit 100 stores an assembly of learning datapieces (plural storage). Learning data is, for example, an image, andeach image includes an object to be a labeling target. An object is, forexample, a person, a dog, a cat, a car, and a building. The learningdata is used to generate a classifier. For example, when learning datais an image capturing a person, a dog, a cat, or the like as an object,and information representing the object is correctly applied thereto asa label, a classifier for identifying an object in image data includingunknown object can be generated by performing machine learning using thelearning data. According to the present exemplary embodiment, an imageis described as an example of learning data, and thus an image issometimes referred to as a learning image.

A label and attribute information storage unit 101 stores an assembly oflabeled results by a labeler to each of the learning data pieces andattribute information pieces accompanying the results. The labeledresult is used as a teaching signal of the machine learning and, forexample, a label “cat” which is applied to an image of a cat by thelabeler. On the other hand, the attribute information includes labelattribute information corresponding to the applied label, labelerattribute information corresponding to the labeler who applies thelabel, and data attribute information as attribute of the learning data.The label attribute information includes an identification (ID)indicating the labeler, a date and time when the labeler performinglabeling, a confidence degree indicating a degree how the labeler canperform labeling correctly. In the case where the labeler is a person,when the person is sure that an object in an image is, for example, acat, a high value is applied as the confidence degree, and when theperson wonders whether the object is a cat or a dog, a low value isapplied as the confidence degree. A value of the confidence degree is,for example, a real number from zero to one, however, the value is notlimited to this example. In the case where the labeler is an algorithm(a classifier), for example, when it is determined whether an object isa cat using a cat classifier on an image, a likelihood representing acat likeness may be used as the confidence degree.

The labeler attribute information is reliability of the labeler. Thereliability is an expected value that the label applied by the labeleris thought to be correct. For example, reliability of a labeler whoalways labels correctly any image is high, and in contrast, reliabilityof a labeler who has a high rate of false labeling is low. The dataattribute information is information inherent in the learning data, forexample, a date and time when the image data is captured, an imagecapturing parameter, a labeling method described below, a determinedlabel, and a certainty degree.

A label and attribute information management unit 110 manages and storesa correspondence relationship between learning data and label andattribute information. Data pieces are stored as a table in a memory notillustrated. FIG. 2 illustrates an example of correspondencerelationships among pieces of data, labels, and attribute information tobe managed. Each learning data in a learning data group is associatedwith the data attribute information corresponding to each data and aplurality of labels. As examples, the data attribute informationincludes a date and time when the data is obtained, parameterinformation of a camera used for capturing an image, and a type of thelabeling method. As examples of labels, an image as learning data 1 isassociated with a plurality of labels namely, “cat” as a label 1 a,“dog” as a label 1 b, and “cat” as a label 1 c. In addition, each labelis imparted with the above-described label attribute informationcorrespondingly. The learning data is associated with a plurality oflabels, and a label is determined like majority voting by a methoddescribed below. Accordingly, effects of an accidental labeling errorand a labeling error by a malicious labeler can be reduced. Every timethe label applied to the learning data and the attribute information ofthe label are received, the label and attribute information managementunit 110 associates the learning data 100 with the label and attributeinformation in the above-described form.

When a request to refer the label corresponding to the learning data isreceived from the outside, a request reception unit 111 makes an inquiryto the label and attribute information management unit 110 and transmitsthe label and the attribute information corresponding to the designatedlearning data (target learning data) to a label determination unit 112.The designated learning data may be one each or a group of a pluralityof data pieces.

The label determination unit 112 determines a likely label based on thelabel and the attribute information corresponding to the learning datareceived from the label and attribute information management unit 110.The method for determining the label is described in detail below.

First, when there is no label corresponding to the learning datareceived by the request reception unit 111, the label determination unit112 returns a replay that a label is unknown. If a label candidate isknown in advance, a label may be randomly selected from the labelcandidates and returned. When there is a plurality of labelscorresponding to the designated learning data, a determined label isselected from the label candidates by scoring. It is regarded that npieces of labels exist with respect to the learning data, and an i-th(i=1, . . . , n) label is an L(i). Further, when the label L(i) includesm types, namely I_1, . . . , I_m, of labels which are different eachother, the label candidate of which a score is the largest in m piecesof the label candidates may be returned as the determined label. Acalculation method (evaluation) of scoring is described. The confidencedegree to labeling, the corresponding labeler, and the reliability ofthe labeler are respectively represented as C(i), A(i), and R(A(i)). Ascore S(j) (j=1, . . . , m) of each label candidate is calculated(derived) by a following formula.

[Formula 1]

S(j)=Σ_(i|L(i)∈l) _(j) ₎ f ₁(C(i))*f ₂(R(A(i)))   (1)

A function f(x) monotonically increases with respect to x and can bevarious functions f(x) as illustrated in FIG. 3 as examples. When thefunction f is adjusted, conditions can be changed such as a score is notgiven to a label of which the reliability of the labeler is a certainvalue or lower, and a label of which the confidence degree to labelingis extremely high is greatly depended. For example, f_1(x)=f_2(x)=x.Further, the score S(j) is calculated for each label candidates I_j, anda label I_j when the score S(j) becomes a maximum value may be regardedas the determined label. As described above, the maximum value of thescore S(j) is selected, and thus a labeler having higher reliability, alabel having a higher confidence degree, and a label with more samelabels are selected as the determined label. Thus, an effect of a labelwhich has a low confidence degree and is highly likely a false label maybe reduced, and if a malicious labeler applies a false label, thereliability of the malicious labeler is lower, so that a negative effecton label determination can be reduced. Further, it is described that theconfidence degree C(i) and the reliability R (A(i)) can be obtained fromthe attribute information in the formula (1), however, the confidencedegree may be applied using the above-described method according to thepresent exemplary embodiment. Regarding evaluation of the reliability(the reliability derivation), for example, a rate of applying a labelsame as the label determined in the past is calculated for each labeler,and the calculated value may be regarded as the reliability, or thelabeler is caused to label a plurality of data pieces of which correctlabels are determined in advance, and a rate of correct answers may beregarded as the reliability. In addition, calculation may be performedby regarding any one of or both of the confidence degree and thereliability are constant values.

However, when no different label (m=1) exists in the label L(i), thereis no need to perform scoring, and a common label L(i)=I_1 may beregarded as the determined label in all “i”.

According to the present exemplary embodiment, a case of a “symboldesignation type” labeling is described in which some sort of names andgood or bad is designated as a label, a scope of application of thepresent disclosure is not limited to this case. For example, the presentdisclosure can be applied to a case of an “area designation type”labeling which is labeling for enclosing a face area in an objectcaptured in an image with a bounding box and labeling for filling a roadarea captured in an image with color. Specifically, a label can bedetermined using a following method.

In the case of the area designation type labeling, True or False (pixellabel) is designated to a certain learning data image as learning data(pixel learning data) independent for each pixel. For example, when aperson area in an image is detected, labeling may be performed in such amanner that a pixel in the area in which the person is captured isdesignated as True, and a pixel in an area other than that is designatedas False. As a method for designating an area, there are, for example, abounding box which designates a rectangular area enclosing a targetperson on an image, and a method for filling pixels in a target personarea with color. When a label is determined from a learning data imagegroup labeled by the above-described method, a pixel label is determinedfor each pixel by the formula (1). FIG. 4 illustrates examples of labeldetermination of the area designation type. FIG. 4 illustrates labeledresults 10 to 17 in which specific areas are filled with color inimages. The pixel label is determined for each pixel with respect to aplurality of the labels. For example, when a third pixel from the lefton a second line from the top is focused, there are seven results 10,11, 12, 14, 15, 16, and 17 in which the relevant pixel is filled withcolor (black) and one result 13 in which the relevant pixel is notfilled with color (white) in FIG. 4. In the formula (1), when theconfidence degree and the reliability are regarded as 1, 0, andf_1=f_2=x regardless of the label for sake of simplicity, the pixellabel can be determined as black since the score S (black)=7, and thescore S (white)=1. When this processing is performed on each pixel, aresult 18 illustrated in FIG. 4 is obtained, and the result 18 can beused as the determined label of the image. In the case of the boundingbox, there is a case in which a determined pixel label representinginside of the bounding box does not have a rectangular shape as a result20 in FIG. 5, however, in such a case, the bounding box may be set bybeing approximated by a rectangular frame as shown in a result 21 inFIG. 5.

In addition, a “numerical value designation type” labeling can beconsidered in which a numeric array such as a matrix and a vectorrepresenting an image or coordinate values and an orientation in athree-dimensional space of a target object is designated as a label. Inthe case of the numerical value designation type label, a numericalvalue of the label can be determined as a weighted average value usingthe score S(j) in the formula (1) as a weight in each label. However, avalue may be determined by, for example, performing robust estimationsuch as M estimation, which is a known technique, in consideration of anoutlier or using random sample consensus (RANSAC), which is a knowntechnique, by assuming some model without being limited to the weightedaverage.

Next, a processing procedure of the information processing apparatus10000 according to the present exemplary embodiment is described withreference to FIGS. 6 and 7. As a major processing flow, labelingprocessing is performed on the learning data, and subsequently,processing for determining a label corresponding to the learning data isperformed. However, it can be returned to the labeling processing afterdetermining the label.

First, labeling processing performed on the learning data is described.FIG. 6 is a flowchart illustrating labeling processing. In step S1001,the label and attribute information management unit 110 obtains a resultthat a labeler applies a label and attribute information to the learningdata and stores the result in the label and attribute informationstorage unit 101. The attribute information may be directly input by thelabeler, and a date and a time when the labeling is performed and an IDof the labeler in the label attribute information may be automaticallyextracted from the information processing apparatus performing labeling.

Next, in step S1002, the label and attribute information management unit110 manages the learning data in association with the label andattribute information in a format as described above in FIG. 2. As aunit to be labeled at a time in steps S1001 and S1002, the label andattribute information management unit 110 may associate the learningdata with each label or associate a plurality of the learning datapieces with a plurality of labels at once without depending on thenumber of the learning data pieces and the number of labels.

Next, in step S1003, the label and attribute information management unit110 checks whether labeling is completed, and when the labeling iscompleted (YES in step S1003), processing for determining the label isready to be performed. When the labeling is not completed (NO in stepS1003), the processing returns to step S1001. A criterion fordetermining whether the labeling is completed is determined whethersufficient labels for determining the label are applied to the learningdata. For example, when x or more pieces of labels are applied to eachlearning data, it may be determined that labeling is completed. However,the criterion is not limited to the above-described one, and it may bedetermined that the labeling is completed when a certainty degreeexceeds a threshold value which is described below in a third exemplaryembodiment.

Each function unit described herein is implemented by the CPU 1810loading a program stored in the ROM 1820 to the RAM 1830 and executingprocessing according to each flowchart described below. Further, forexample, when hardware is constituted as an alternative to softwareprocessing using the CPU 1810, an arithmetic unit and a circuit may beconstituted so as to correspond to processing of each function unitdescribed herein.

FIG. 7 is a flowchart illustrating a processing flow by the informationprocessing apparatus 10000 according to the present exemplaryembodiment. In step S1101, the request reception unit 111 receives arequest from a user and a system. A content of the request is, forexample, “return a label corresponding to certain learning data”. Whenreceiving the request, the request reception unit 111 transmits an ID ofthe designated learning data to the label and attribute informationmanagement unit 110 and requests the label thereof. In this regard, thelearning data to be requested is not limited to one and may be aplurality of pieces, and in this case, a plurality of IDs of thelearning data pieces is transmitted to the label and attributeinformation management unit 110 correspondingly.

Next, in step S1102, the label and attribute information management unit110 determines a label and attribute information corresponding to thelearning data ID received from the request reception unit 111 andtransmits the label and the attribute information to the labeldetermination unit 112.

Next, in step S1103, the label determination unit 112 performs scoringon the label candidates based on the formula (1) and determines a label.

In step S1104, the label determination unit 112 returns the determinedlabel to the user and the system which made the request.

The present exemplary embodiment is not limited to perform theprocessing in the order described above in FIG. 7. For example, thelabel and attribute information management unit 110 can transmit thelabel and the attribute information corresponding to the learning datain advance to the label determination unit 112, and the labeldetermination unit 112 can determine the label and store the determinedlabel in the data attribute information before receiving a request. Inthis case, when the request reception unit 111 requests a label in stepS1101, the label and attribute information management unit 110 maytransmit the determined label stored in the data attribute informationto the label determination unit 112, and the label determination unit112 may transmit the determined label as it is to the user and thesystem which made the request without performing calculation processing.

According to the present exemplary embodiment, the label and attributeinformation management unit manages a plurality of labels and variousattribute information pieces with respect to each learning data, and thelabel determination unit appropriately determines a label form theseinformation pieces, so that, if a part of labels includes an error, acorrect label can be determined by suppressing an effect of the error.

An information processing apparatus 20000 according to a secondexemplary embodiment of the present disclosure presents, in a case wherea labeler is a user and when the user performs labeling, a label itselfapplied by another user and a similarity degree of a label which theuser currently intends to apply and enables the user who performslabeling to efficiently perform labeling. In other words, according tothe second exemplary embodiment, when a label applied by another useralready exists in certain learning data, the already existing label iscompared with a label and attribute information to facilitate labeling.

FIG. 8 is a functional block diagram illustrating a configuration of alabeling apparatus 20000 according to the second exemplary embodiment.The configuration of the labeling apparatus 20000 according to thesecond exemplary embodiment includes portions in common with the exampleof the configuration according to the first exemplary embodimentillustrated in FIG. 1, so that a request reception unit 211, a labelcomparison unit 113, and a display control unit 114 which are differentfrom the first exemplary embodiment are described. Further, theinformation processing apparatus according to the second exemplaryembodiment is connected to a display apparatus 30 in a wired way orwirelessly. The display apparatus may adopt any display format, and thepresent exemplary embodiment is described as using a touch panel typedisplay. However, an organic electroluminescence (EL) display, a cathoderay tube (CRT) display, and a projector which displays an image byprojecting it on a wall and the like may be used as the displayapparatus. In addition, a keyboard and a mouse may be used as an inputdevice to the display apparatus instead of the touch panel.

The request reception unit 211 receives a request regarding a label tobe referred to from a user and transmits the request to the label andattribute information management unit 110 in addition to the functionaccording to the first exemplary embodiment. The label and attributeinformation management unit 110 transmits information pieces such as thelabel and the attribute information to the label comparison unit 113 andthe display control unit 114 upon receiving an order from the requestreception unit 211. The label comparison unit 113 receives two pairs ofrelated labels from the label and attribute information management unit110, calculates a similarity degree between labels, and transmits alabel similarity to the display control unit 114. The display controlunit 114 receives the label and the attribute information from the labeland attribute information management unit 110, receives the labelsimilarity degree from the label comparison unit 113, and causes, forexample, the display apparatus 30 to display based on these informationpieces.

As a user request that the request reception unit 211 receives from theuser, for example, when the request is to “want to refer a label appliedto certain learning data by each labeler”, the label and attributeinformation management unit 110 extract a label in the designatedlearning data and transmits the label to the display control unit 114. Acontent to be displayed on the display apparatus 30 by the displaycontrol unit 114 is described in detail below.

The label comparison unit 113 calculates a similarity degree between twolabels (a label A and a label B). First, calculation of a similaritydegree in the symbol designation type labeling is described. When thelabel A is “cat”, and the label B is also “cat”, the label A and thelabel B are the same, and thus a similarity degree is 1.0. When thelabel A is “cat”, and the label B is a “dog”, the label A and the labelB are different, and thus the similarity degree is 0. Further, forexample, when a plurality of labels for enumerating objects captured inan image is applied, a similarity degree is calculated as (the number oflabels common to the label A and the label B)/(the number of labelsincluded in either one of the label A and the label B). For example,when the label A is “dog, cat, and monkey”, and the label B is “dog,cat, horse, and cow”, a similarity degree is (dog and cat: 2)/(dog, cat,monkey, horse, and cow: 5)=0.4. However, the calculation method of thesimilarity degree is not limited to the above-described formula, and amethod can be adopted as long as a similarity degree between two labelsis a higher value as labels are similar and is a lower value as labelsare different. For example, when it is assumed that the label A is alabel being edited by a user, and the label B is a label applied byanother labeler, the label comparison unit 113 calculates a similaritydegree and a coincidence degree therebetween, and the user can performlabeling while considering whether the label being edited is a likelylabel by referring to the similarity degree and the coincidence degreeby a user interface (UI) described below.

The label comparison unit 113 can calculate not only a similarity degreebut also a coincidence degree of the label. A coincidence degree is anindex indicating how much a label applied by the user coincides with aplurality of labels applied by another labeler. More specifically, thecoincidence degree can be obtained by (a total sum of a similaritydegree of the labels applied by the user and the labels applied by theother labeler)/(the number of labels applied by the other labeler) withrespect to certain learning data. As a method for selecting label groupswith respect to the certain learning data, for example, all labelsassociated with the learning data may be selected. However, theselecting method is not limited to the above-described one, and labelsof a labeler whose reliability is low and a label group excluding labelsof which a confidence degree is low may be used, and the user mayarbitrarily select a label group. The coincidence degree can be referredto by the display control unit 114 as with the similarity degree. Thecoincidence degree is calculated and referred to, and thus an error inlabeling caused by a mistake and a malfunction can be reduced.

On the other hand, in calculation of the similarity degree in the areadesignation type labeling, the similarity degree is obtained from a ratewith respect to the number of pixels based on whether a pixel label isthe same or not for each pixel. A term w represents that a certain pixelis within the bounding box or filled with color, and a term b representsthat a certain pixel is out of the bounding box or not filled withcolor. In a designated learning data image, a variable Nwb representsthe number of pixels which has the label A and is classified as “w” andpixels which has the label B and is classified as “b”, and variablesNww, Nbb, and Nbw are defined in a similar manner. A similarity degreeRe of the label A and the label B is expressed by a following formula.

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 2} \rbrack & \; \\{{Re} = \frac{N_{ww} + N_{bb}}{N_{ww} + N_{bb} + N_{wb} + N_{bw}}} & (2)\end{matrix}$

When an inner area of the bounding box in an image is small, or an areafilled with color is small, the variable Nww becomes predominantly largecompared to other variables, and thus the similarity degree Re is veryclose to 1.0, and a difference is very small. In such a case, thesimilarity degree Re is calculated using the formula (2) by overwritingthe value of Nww with the small value (for example, Nww=0), and thus thesimilarity degree can be calculated by weighting the designated areaeven the area is small.

Regarding calculation of the similarity degree in the numerical valuedesignation type labeling, a difference d between values of the label Aand the label B is calculated, a monotonically decreasing function gwhich is inverse to the function f in FIG. 3 is considered whichapproaches 1.0 when the difference d is zero and approaches 0.0 when thedifference d becomes larger, and the similarity degree may be defined byg(d). The value d is defined as a difference between values of the labelA and the label B here, however, the present exemplary embodiment is notlimited to this value, and, for example, a L1 norm and a L2 norm may beused as the value d in the case that the label A and the label B arevectors and matrices.

Regarding calculation methods of the coincidence degrees in the areadesignation type and the numerical value designation type labeling, thedescriptions thereof are omitted since the coincidence degrees can becalculated similarly by the method described in the symbol designationtype after respectively calculating the similarity degrees.

The display control unit 114 receives necessary information from thelabel and attribute information management unit 110 and the labelcomparison unit 113 and performs display on the display apparatus 30which is a touch panel type display. An external appearance and afunction of the display apparatus 30 are described with reference toFIG. 9. FIG. 9 is an example of display for editing the symboldesignation type label.

The display apparatus 30 is the touch panel type display and includes atouch panel liquid crystal display 31, labeling target image data 32, alabel input portion 33 for the target image data, a label list 34 of thetarget image data, a thumbnail image data display area 35, a displaycondition setting portion 36, a similarity degree display portion 37, alabeling type switching button 38, a temporary storing button 39, animage switching button 40, a backward and forward button 41, a labeltransfer button 42, a confirmation button 43, a label determinationbutton 44, and a label comparison button 45.

The display apparatus 30 has a function of displaying a contentinstructed by the display control unit 114 and a function of receiving arequest received by the request reception unit 111. The displayapparatus 30 may include functions of the label and attributeinformation management unit 110, the label determination unit 112, andthe label comparison unit 113, or an external information processingapparatus includes these functions, and the display apparatus 30 mayexchange information therewith via a communication unit.

The touch panel liquid crystal display 31 displays an image, a button,and the like. The touch panel liquid crystal display is described here,however, the display may not be a touch panel type, and in such a case,for example, a mouse may be used to perform a button operation and thelike.

In the labeling target image data 32, an image of a target that a usercurrently labels is displayed. An arbitrary area in the displayed imagecan be enlarged or reduced by a user operation.

The label input portion 33 for the target image data is an area forediting a label regarding the image displayed on the labeling targetimage data 32. The label may be directly input with characters using akeyboard and the like or selected by a button or from a list whencandidates to be selected as a label are determined. When labeling isperformed on image data on which a user himself/herself has alreadylabeled, the label that the user applied in the past can be displayed onthe label input portion 33.

The label list 34 of the target image data is an area for displaying alist of labels already applied to the target image by the userhimself/herself of another labeler. The user can think a label of thetarget image with reference to the label list 34. In addition, when thedetermined label has been obtained, the determined label is included inthe label list 34.

In the thumbnail image data display area 35, a thumbnail of the learningdata image is displayed. FIG. 9 illustrates an example in which fourthumbnails are displayed, however the display is not limited to thisexample, and the display can be switched to a larger image, a smallerimage, a file name, and the like. Further, the thumbnail image datadisplay area 35 can be scrolled, and the user can browse another imageby, for example, touching the display area with a finger or the like andsliding it to right and left. In addition, the number of applied labelsand whether the determined label is registered to the data attributeinformation can be displayed as the thumbnails, and the user can selectan image to be labeled next from these information pieces.

The display condition setting portion 36 is used to set a label to bedisplayed in the label list 34 or a condition of an image to bedisplayed on the thumbnail image data display area 35. Which conditionis set can be switched in the display condition setting portion 36. Asan example of the condition, the above-described “want to refer a labelapplied to certain learning data by each labeler” may be included. In amode for setting a condition of the label list 34, for example, acondition for displaying only labels of which the confidence degree isgreater than or equal to 0.8 and a condition for arranging thereliability of the labeler in descending order in the label list are setbased on the label attribute information and the labeler attributeinformation and reflected to the label list 34. On the other hand, in amode for setting a condition of the thumbnail image data display area35, for example, a condition for displaying an image captured in 2016 orlater and a condition for displaying only images of which labeledresults are less than five pieces are set based on the data attributeinformation and the label attribute information, and thus a thumbnailimage to be a target can be displayed on the thumbnail image datadisplay area 35. As described above, the user refers to the image datapieces and labeled results corresponding to the set condition whilefiltering and sorting them and accordingly can improve labeling accuracyby more specifically visualizing a labeling rule. For example, thelabeled results are displayed in descending order of the labelerreliability with respect to the same the image data, and thus the usercan confirm a tendency of a likely label and improve the confidencedegree of labeling. Further, a group of images applied with the samelabel is displayed as a thumbnail, and accordingly the user canunderstand a tendency of the images to be applied with the relevantlabel and can perform more accurate labeling. In addition, the userrefers to the label that the user himself or herself applied before andcan confirm that whether a labeling rule of himself or herself iswavered or not. In addition, a group of images labeled before a datewhen the labeling rule is changed is selected by performing filtering,and accordingly a label can be efficiently applied again only to theimage group on which a new rule is not applied.

The similarity degree display portion 37 displays a similarity degreebetween a label set in the label input portion 33 and a label selectedfrom the label list 34. The similarity degree is calculated by theabove-described label comparison unit 113. In addition, the similaritydegree display portion 37 can switch display of the similarity degreeand display of the coincidence degree by a coincidence degree switchingbutton, which is not illustrated. The coincidence degree is calculatedby the above-described label comparison unit 113.

The labeling type switching button 38 is a button for switching the typeof the labeling defined by a user, such as the symbol designation typelabeling, the area designation type labeling, and the numerical valuedesignation type labeling. When the labeling type is switched, an imagedata group to be a target is also changed, so that contents and the likein the labeling target image data 32, the label list 34, and thethumbnail image data display area 35 are updated. The case of switchingthe labeling type is described as an example, however, the presentexemplary embodiment is not limited to this example. The image datagroups can be switched, and when the image data group is designated, thelabeling type may be updated according to the type of the labelingmethod in the data attribute information.

The temporary storing button 39 is a button for temporarily storing thecontent in the label input portion 33. The content is stored, and thusin the case that the display is switched to other image data and thenreturned to the previous image, the label can be edited again from thestored state. However, the temporary storing button 39 is notnecessarily in a button form, and the content may be alwaysautomatically stored when the label input portion 33 is edited.

The image switching button 40 is used to switch the content of thelabeling target image data 32. The content may be switched bydesignating a thumbnail image displayed on the thumbnail image datadisplay area 35, or images may be randomly switched without designation.

The backward and forward button 41 is used to return to a previoushistory state by the backward button or to proceed to a next historystate, if there is the next history state, by the forward button from ahistory of operation performed by a user.

The label transfer button 42 is used to select a label from the labellist 34 and transfer the label same as the selected label to the labelinput portion 33.

The confirmation button 43 is used to determine the label written in thelabel input portion 33 and registers the label to the label andattribute information. When the confirmation button 43 is pressed, ascreen for setting the confidence degree to the labeling by the user isdisplayed, and the confidence degree is stored as the attributeinformation at the same time. The confirmation button 43 is also used todisplay a message for prompting the user to confirm that there is noerror in the label and an option of whether to edit the label in thecase where a similarity degree between a label newly applied by the userand the determined label is low, or the newly applied label is greatlydifferent from the tendency of the labels applied by the user himself orherself in the past. Accordingly, the user can reduce a possibility ofregistering a false label due to a malfunction and a mistake.

The label determination button 44 is used to determine the label usingthe label determination unit 112 based on a label group corresponding tothe designated image data. When the label is determined, the determinedlabel is added to or updated in the label list 34.

The label comparison button 45 is used to compare the label regardingthe labeling target image data 32 being edited in the label inputportion 33 with the label designated in the label list 34 using thelabel comparison unit 113 and calculate the similarity degreetherebetween. The calculated similarity degree is displayed on thesimilarity degree display portion 37. The label comparison button 45 isdescribed here, however, the present exemplary embodiment is not limitedto this button, and label comparison and calculation of the similaritydegree may be automatically performed every time the label is designatedin the label list 34, and the similarity degree display portion 37 maybe updated.

Next, FIG. 10 illustrates an example of display for editing the areadesignation type label on the touch panel type display. Many functionsin FIG. 10 are same as those in FIG. 9, and thus different functions areonly described.

FIG. 10 includes comparison target labeling image data 46. A user canselect the label already applied to the same image in the labelingtarget image data 32 and display the label by overlapping with the imagedata.

In the labeling target image data 32, an image which is a current targetof labeling by the user is displayed as with the symbol designationtype, however, a function of labeling by designating an area on theimage is added. Accordingly, the processing performed on the label inputportion 33 in FIG. 9 is integrated as the function of the labelingtarget image data 32.

As a thumbnail image displayed on the thumbnail image data display area35, not only thumbnail display of the learning data image as with thesymbol designation type but also a thumbnail of an image in which alabeled result is overlapped on the labeling target image data can bedisplayed. Accordingly, in the display condition setting portion 36, theuser can further set a condition by designating a category of an imageon which the labeled result is overlapped.

The similarity degree display portion 37 displays the similarity degreebetween the label set in the labeling target image data 32 and the labelin the comparison target labeling image data 46.

The label transfer button 42 transfers the label displayed in thecomparison target labeling image data 46 as the label being edited inthe labeling target image data 32. Accordingly, the user can efficientlyselect the label from labels applied in the past, edit and apply thelabel.

It is described above that various buttons are displayed on the touchpanel liquid crystal display 31, however, a processing content may beselected from a list or assigned to a shortcut key instead of thebuttons without being limited to the above-described configuration.

An example of display for editing the numerical value designation typelabel is similar to that of the symbol designation type excepting apoint that a numerical value is input as a label instead of selectingsymbols as character input in the example of the symbol designation typeillustrated in FIG. 9, so that description thereof is omitted.

Next, FIG. 11 illustrates a typical flow for applying and confirming thelabel using the information processing apparatus according to thepresent exemplary embodiment. Processing flows are approximately similarin any of the symbol designation type, the area designation type, andthe numerical value designation type labeling and thus describedtogether with reference to FIG. 11. In step S2001, when a user starts upthe display apparatus 30, initialization processing is performed, andthe display control unit 114 displays a UI for labeling as illustratedin FIG. 9 or FIG. 10. The initialization processing includes processingfor reading necessary data pieces from the learning data and the labeland attribute information.

Next, in step S2002, the label and attribute information management unit110 selects image data to be labeled by the user. In the selection, theuser designates the image using the display condition setting portion 36and the thumbnail image data display area 35 and presses the imageswitching button 40 to display the image on the labeling target imagedata 32. Alternatively, the user may randomly display images by pressingthe image switching button 40 without designating the image.

Next, in step S2003, the label and attribute information management unit110 receives the labeled result by the user. The user performs thelabeling operation using the temporary storing button 39, the backwardand forward button 41, the label transfer button 42, and the like.

Next, in step S2004, the label comparison unit 113 compares and confirmsthe labels. The label list 34 or the comparison target labeling imagedata 46 is used as previous labeled results in the comparison andconfirmation. A calculation result of the similarity degree is displayedon the similarity degree display portion 37, and the user can confirm adifference from the label of the comparison target while watching thedisplay. Regarding filtering and sorting of the data, the displaycondition setting portion 36 and thumbnail images may be used.

Next, in step S2005, the label and attribute information management unit110 confirms whether labeling is completed. When the labeling is not yetcompleted (NO in step S2005), the processing returns to step S2003. Whenthe labeling is completed (YES in step S2005), the processing proceedsto step S2006. When confirming completion of the labeling, the userpresses the confirmation button 43 to confirm that there is nopossibility of an error in the label.

Next, in step S2006, the label and attribute information management unit110 registers and manages the image data by adding the attributeinformation such as the confidence degree of the user to the labeling.When the image data is registered, the label and attribute informationmanagement unit 110 associates the learning data with the label and theattribute information.

Next, in step S2007, the user determines whether to continue labeling onanother image. When labeling is continued (YES in step S2007), theprocessing returns to step S2002. When labeling is not continued (NO instep S2007), the processing is terminated.

According to the present exemplary embodiment, a user can efficientlyperform labeling while visually confirming another label and image datausing a label similarity degree calculation function of the labelcomparison unit 113 and functions of comparing, referring, and editing alabel by the UI using the display apparatus.

An information processing apparatus 30000 according to a third exemplaryembodiment causes many labelers to perform labeling by crowdsourcing anddetermines a label with a high degree of accuracy from a large amount oflabels. Further, the information processing apparatus 30000appropriately extracts learning data to be processed by the labeler bycalculating data of which a label is not determined in the learning datapieces and thus efficiently labels a large amount of learning datapieces.

FIG. 12 is a functional block diagram illustrating a functionalconfiguration of a labeling apparatus 30000 based on crowdsourcingaccording to the third exemplary embodiment. The functionalconfiguration according to the third exemplary embodiment includesportions in common with the configuration according to the secondexemplary embodiment illustrated in FIG. 8, so that crowdsourcing 102, acertainty degree calculation unit 115, and a data extraction unit 116which are different from the second exemplary embodiment are describedin detail.

The crowdsourcing 102 is constituted of many labelers who performlabeling, each labeler performs labeling on a learning data groupextracted by the data extraction unit 116, and the label and attributeinformation management unit 110 stores and manages the labeled resultsin the label and attribute information. However, the third exemplaryembodiment is not necessarily limited to the crowdsourcing 102 and mayadopt outsourcing with particular people or cause a plurality ofpersonal computers (PCs) to perform labeling in parallel.

The certainty degree calculation unit (certainty degree derivation unit)115 receives the label and attribute information of each learning datafrom the label and attribute information management unit 110 andcalculates a certainty degree as a degree of likelihood of a labeldetermined by the label determination unit 112. The calculated certaintydegree is stored as the data attribute information for each learningdata. As the certainty degree is higher, the label determined by thelabel determination unit 112 has high likelihood which represents thatno further labeling is necessary. When the certainty degree is low, itindicates that the label is highly likely wrong even the labeldetermined by the label determination unit 112, and thus it is necessaryto perform labeling by more labelers. Or, there is a possibility thatthe learning data itself is data to which a label is difficult to beuniquely determined.

The data extraction unit 116 extracts the learning data to be labeled bythe labeler in the crowdsourcing 102 based on the certainty degree inthe data attribute information for each learning data. Since labeling isless necessary for the learning data of which the certainty degree ishigh, the learning data is not easily extracted as data. In contrast,the learning data of which the certainty degree is low needs to improvethe certainty degree by performing labeling and thus is easily extractedas data. The extracted learning data group is transferred to thecrowdsourcing 102 to be labeled.

Next, a calculation method of a certainty degree (certainty degreederivation) by the certainty degree calculation unit 115 is described indetail below. A certainty degree F. is calculated by a following formulausing the score S(j) of each label candidate calculated by the formula(1).

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 3} \rbrack & \; \\{F = {\frac{\max\limits_{j}\; {S(j)}}{\sum\limits_{j}{S(j)}}*{W(n)}}} & (3)\end{matrix}$

In the formula (3), a function W(n) determines a maximum value of thecertainty degree F. in response to the number of labels. When a value nis small, the function W(n) will also be a small value, and when thevalue n is large, the function W(n) will also be a large value. However,an upper limit of the function w(n) is 1.0. The function W(n) isprovided as described above so as not to easily output the highcertainty degree F. when the total number of labels is small (when thevalue n is small). However, the calculation of the certainty degree isnot limited to the above-described formula as long as a formula uses afact that a ratio of the determined label is high. For example, afollowing formula may be used instead of the formula (3).

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 4} \rbrack & \; \\{F = {\frac{{S\lbrack 1\rbrack} - {S\lbrack 2\rbrack}}{S\lbrack 1\rbrack}*{W(n)}}} & (4)\end{matrix}$

In the formula (4), a value S[x] is an x-th value when the score S(j)(j=1, . . . , m) is arranged in descending order. In other words, avalue S[1]=the max S(j), and a value S[m]=the min S(j). The certaintydegree F. is calculated for each learning data as described above.However, when j>m, the score S[j]=0.

Next, a data extraction method by the data extraction unit 116 isdescribed. Extraction of data is based on following three basicpolicies. (i) Data of which a certainty degree is low is extracted. (ii)Data of which the certainty degree remains low even the number of labelsis increased is regarded as an exception. (iii) Test data to be used forevaluating reliability of a labeler is extracted. Each policy isdescribed.

The policy (i) is because that as the certainty degree is lower, thedata is subjected to labeling by a next labeler from a data group in thelearning data so as to make a more correct label easier to be selected.The policy (ii) indicates that the learning data itself has a defect.The fact that the certainty degree is not increased even the number oflabels is increased for certain learning data indicates that a pluralityof peaks or no peak of the score S(j) exists in distribution of thescore S(j) by the formula (3) or the formula (4). In such a case, it ishighly likely that the learning data itself is an image difficult to belabeled, for example, a case is considered in which a target object isonly partly captured in an image and indistinguishable to be labeled. Inthis case, an attribute for indicating that the data is difficult to belabeled is defined and registered to the data attribute information.Thus, data extraction is performed so as to make a request to a labelerhaving high reliability who can highly accurately perform labeling forlabeling of the learning data which is difficult to be labeled, andaccordingly an error in labeling can be reduced. In addition, thelearning data which is difficult to be labeled may be excluded fromtarget learning data to maintain a quality of the learning data group.The policy (iii) means a reliability evaluation data set for testing thereliability of the labeler. A data set for evaluating the reliability isprepared in advance which is constituted of data of which a true labelis known and a data group with the high certainty degree, and thereliability evaluation data set is partly included in data pieces to belabeled by the labeler without the labeler's noticing. Subsequently, thereliability of the labeler can be obtained by evaluating how the labelsapplied to the reliability evaluation data set by the labeler arecorrect compared to the true labels. One of points requiring attentionto crowdsourcing is incorporation of a false label by a malicious user.Using the reliability evaluation data set lowers the reliability of thelabeler who applies a false label, so that if such a malicious user isincluded, an effect thereof can be suppressed. Further, when theinitialization is performed (when labeling is started), the label andattribute information to the learning data is not defined at all, andthe certainty degree is not set and zero with respect to all learningdata pieces, so that the data extraction unit 116 may transfer dataextracted by appropriately dividing all of the learning data pieces tothe crowdsourcing 102. However, the behavior in the initialization isnot limited to the above-describe one. At the time of initialization,some data pieces to be a model of labeling may be registered in advance,and data may be extracted using the data pieces as first label andattribute information and reliability evaluation data set.

Next, a processing procedure of the information processing apparatus30000 according to the present exemplary embodiment is described. FIG.13 is a flowchart illustrating a flow of labeling based on thecrowdsourcing. In step S3001, the request reception unit 211 receives arequest to perform a labeling operation from the labeler in thecrowdsourcing 102.

Next, in step S3002, the data extraction unit 116 extracts data to belabeled. Data may be extracted one by one or in a plurality of datagroups together.

Next, in step S3003, the extracted data is transferred to the labeler inthe crowdsourcing 102 to perform labeling. The labeling procedure may beperformed by following the flowchart already described in FIG. 11.

Next, in step S3004, the label and attribute information management unit110 obtains the label and the attribute information from thecrowdsourcing 102 and manages them in association with the learningdata. A detail processing flow in step S3004 may be performed byfollowing the flowchart described in FIG. 6.

Next, in step S3005, the certainty degree is calculated for eachlearning data using the certainty degree calculation unit 115. Thecalculated certainty degree is updated as the data attribute informationwith respect to the learning data.

Next, in step S3006, the label is determined using the labeldetermination unit 112. The label determination flow may be performed byfollowing the flowchart already described in FIG. 7.

The processing is described to be performed in the order illustrated inthe flowchart in FIG. 13, however the order is not limited to theabove-described one, and, for example, the determination of the label instep S3006 is calculated using the score S(j) in the formula (1) as withthe certainty degree in step S3005, so that the label may be determinedat the same time when the certainty degree is calculated. In addition,some use cases require the determined label if the label is notnecessarily determined. In such a case, the processing in the flowchartin FIG. 7 may be executed at an arbitrary timing independent from theprocessing in the flowchart in FIG. 13 using the certainty degree.

According to the present exemplary embodiment, the certainty degreeindicating whether the label is determined is calculated, and data isextracted from data of which the label is not determined, so that thelearning data which is necessary to be labeled can be efficientlyselected and transferred to the cloud. In addition, the reliabilityevaluation data set is included in the data to be extracted, andaccordingly a harmful effect such as incorporation of a labeling errorby a malicious user in the crowdsourcing can be minimized.

An information processing apparatus 40000 according to a fourthexemplary embodiment determines a label with fewer errors from a largeamount of labels by crowdsourcing, performs learning using the labelwith fewer errors, and thus can highly accurately perform recognition.

FIG. 14 is a functional block diagram illustrating a functionalconfiguration of the information processing apparatus 40000 according tothe fourth exemplary embodiment. The functional configuration accordingto the fourth exemplary embodiment includes portions in common with theexample of the configuration according to the third exemplary embodimentillustrated in FIG. 12, so that a learning unit 117 and a recognitionunit 118 which are different from the third exemplary embodiment aredescribed in detail.

The learning unit 117 receives the label determined by the labeldetermination unit 112 and the learning data by making a request to therequest reception unit 211 and performs supervised learning using thelabel corresponding to the learning data as a teacher. The learningmethod is not particularly limited, and deep learning and random forestmay be used. As data to be used in the learning, only data of which thecertainty degree is greater than or equal to a threshold value is used.Accordingly, the learning can be performed by labels with fewer errors,and a highly accurate prediction model can be generated. However, datais not limited to the above-described one, and the learning may beperformed using all of the learning data pieces. The learning can beperformed at an arbitrary timing if the determined label correspondingto the learning data can be obtained.

When receiving the prediction model from the learning unit 117 and thequery data from the request reception unit 211, the recognition unit 118returns a predicted result (an output label) based on the predictionmodel. The output label is used for comparison with another label in thelabel comparison unit 113, displayed by the display control unit 114with the other label, and referred to when the certainty degreecalculation unit 115 determines the certainty degree. For example, thedisplay control unit 114 causes the display apparatus 30 to displaycomparison with the output label during editing of the label and thuscan prompt a user to correct the label.

Next, a calculation method of the certainty degree when the output labelis obtained by the certainty degree calculation unit 115 is described.Regarding the calculation method, the certainty degree F. is alreadycalculated by the formula (3) or the formula (4). When the output labelis known, a formula B (I_d, I_o) is further added to the formula. Inthis regard, “I_d” and “I_o” respectively represent the determined labeland the output label. The formula B indicates a similarity degree with alarger value when “I_d” and “I_o” are the same and a similarity degreewith a smaller value when “I_d” and “I_o” are different. According tothe present exemplary embodiment, it is described that the formula B(I_d, I_o) is added to the formula (3) or the formula (4) for thecalculation of the certainty degree. However, the calculation method isnot limited to this, and the certainty degree may be calculated bymultiplying the formula (3) or the formula (4) by the formula B (I_d,I_o). In addition, it is described that the certainty degree calculationunit 115 determines the determined label after calculating the certaintydegree with reference to FIG. 13. In this regards, the label may bedetermined as a tentatively determined label before the certainty degreeis determined, and when the tentatively determined label is equal to theoutput label, and the certainty degree is high, the tentativelydetermined label may be finally regarded as the determined label.

Next, a processing procedure of the information processing apparatus40000 according to the present exemplary embodiment is described. FIG.15 illustrates a flow of learning and recognition by the crowdsourcing.In step S4001, the label determination unit 112 determines the labelusing the flow illustrated in FIG. 13 as with the third exemplaryembodiment. In step S4001, labeling is completed to a stage at which thelabel of data to be subjected to learning can be determined in thelearning data. In addition, it is confirmed not only that the determinedlabel is applied to the learning data, but also the certainty degree ofeach learning data is greater than or equal to a constant valuedepending on a condition.

Next, in step S4002, the learning unit 117 performs the supervisedlearning using the determined label corresponding to the learning dataand estimates the prediction model.

Next, in step S4003, the request reception unit 211 receives the querydata and transmits the query data to the recognition unit 118. The querydata has a format similar to that of the learning data and is, forexample, an image file. The image file is obtained by an imageobtainment unit, which is not illustrated. Step S4003 is an event whichcan occur at an arbitrary timing when the prediction model is generatedand not necessarily to be executed after step S4002 once the predictionmodel is generated in step S4002.

Next, in step S4004, the recognition unit 118 predicts (recognizes) aresult with respect to the query data based on the prediction modelobtained in the previous processing. The predicted result has a formsimilar to that of the label. For example, in the case of the symboldesignation type, the recognition unit 118 predicts (recognizes) that ananimal captured in the query data is a “cat”, and in the case of thearea designation type, the recognition unit 118 predicts, for example, aresult in which only a face area of an object is filled with color. Inaddition, in the case of the numerical value designation type, therecognition unit 118 predicts a matrix and quaternion indicating aposture of a target object. The predicted (recognized) result may bereturned to use the recognition unit 118 as a recognition system or toimprove the calculation in the certainty degree calculation unit 115.Further, the predicted (recognized) result may be used as a comparisontarget in the display control unit 114.

The learning unit 117 and the recognition unit 118 are describedaccording to the present exemplary embodiment having the configurationin which the learning unit 117 and the recognition unit 118 are added tothe information processing apparatus according to the third exemplaryembodiment, however the configuration is not limited to this, and thelearning unit 117 and the recognition unit 118 may be included in thefirst exemplary embodiment and the second exemplary embodiment in thesimilar manner.

According to the present exemplary embodiment, a prediction model isgenerated by performing learning using labels with fewer errorsdetermined by the label determination unit, and the prediction model isused to perform recognition, so that the recognition can be highlyaccurately performed. Further, a certainty degree is calculated with ahigh degree of accuracy using the recognition result, and accordingly, amore reliable determined label can be obtained, and accuracy in dataextraction can be improved. In addition, the display control unit 114displays the recognition result to be compared with a label beingedited, and thus a labeler can be helped to reduce an error in labeling.

An information processing apparatus 50000 according to a fifth exemplaryembodiment has a function of comparing a label applied by a labeler withan existing label and evaluating a reliability of the labeler.Evaluating the labeler with a high degree of accuracy leads toappropriately pay a reward for a labeling operation and improvemotivation and work efficiency of the labeler himself or herself. Inaddition, a malicious labeler who intends to apply a false label can beidentified.

FIG. 16 is a block diagram illustrating an example of a configuration ofthe information processing apparatus 50000 according to the fifthexemplary embodiment. The example of the configuration according to thefifth exemplary embodiment includes portions in common with the exampleof the configuration according to the second exemplary embodimentillustrated in FIG. 8, so that a labeler evaluation unit (labelevaluation unit) 119 which is different from the second exemplaryembodiment is described. The information processing apparatus 50000according to the fifth exemplary embodiment does not include the displaycontrol unit 114 compared to the information processing apparatus 20000according to the second exemplary embodiment illustrated in FIG. 8,however an evaluation result by the labeler evaluation unit 119 may bedisplayed on the display apparatus 30 via the display control unit 114.

The labeler evaluation unit 119 compares a label newly applied by alabeler with the determined label or the output label and evaluates thenewly applied label. Further, the labeler evaluation unit 119 evaluatesthe labeler based on an evaluation with respect to an entire label groupapplied by the labeler. When the labeler is evaluated, the reliabilityof the labeler can be calculated more accurately, and accordingly, anerror in the determined label can be reduced. In addition, theevaluation can be used as a factor for determining a reward to thelabeler in the crowdsourcing 102.

First, a label evaluation method is described. The label newly appliedby the labeler is evaluated by calculating a similarity degree using thelabel comparison unit 113 with respect to the determined labelcalculated by the label determination unit 112 or the output labelcalculated using the prediction model by the recognition unit 118. Acalculation method of the similarity degree is the same as thatdescribed according to the second exemplary embodiment. An evaluationvalue is determined in a range from zero to one so that the evaluationbecomes higher as the similarity degree is higher such as the evaluationvalue equal to the similarity degree. It is considered that thedetermined label and the output label are ideally the same label, thesimilarity degree may be calculated with respect to any of them, or thesimilarity degree may be calculated to both of them, and an averagevalue of the similarity degrees may be regarded as the evaluation value.However, when the determined label and the output label are widelydifferent from each other, it is possible that the attribute informationincludes an inadequate setting in the calculation of the formula (1) bythe label determination unit 112, and the learning of the predictionmodel by the learning unit 117 has failed. In such a case, the newlyapplied label cannot be correctly evaluated, so that the evaluation ofthe label to the learning data is invalidated.

Next, a labeler evaluation method is described. A reward parameter Rwand a reliability R of a labeler are calculated. The label applied bythe labeler using the above-described method is given theabove-described evaluation value v(k) (k=1, . . . , p) based on thesimilarity degree for each learning data expecting the label of whichthe evaluation is invalidated. Further, the reward parameter Rw of thelabeler is calculated by a following formula.

Rw=Σkf(r)*v(k)   (5)

The formula (5) includes the monotonically increasing function f and thereliability R of the labeler used in the formula (1). For example,f(R)=1+R. Further, the reliability R can be calculated by a formula (6)described below. A reward of the labeler in the crowdsourcing isdetermined based on the reward parameter Rw of the labeler. During whenthe labeling is continuously performed on the learning data, thedetermined label, the prediction model, and the like change at any time,so that if the reward parameter Rw is calculated in that stage, thecalculated parameter is not always correct. Therefore, the rewardparameter Rw can be defined from a label applied to data regarded aslabeling is completed with respect to the learning data. However, thereliability R of the labeler at that time is used. The reward parameterRw may be added for each data on which labeling is completed one afteranother as shown in the formula (5). Next, the reliability R of thelabeler is calculated by a following formula.

R=W(p)*Πkv(k)   (6)

In the formula (6), a function W(p) is similar to the function W(n) inthe formula (3) and has a value from zero to one for determining amaximum value of the reliability R according to the number p of theevaluation values. The calculated reliability R of the labeler isupdated as the labeler attribute information. When the reliability R ofthe labeler is less than a threshold value, the labeler is recognized asa malicious labeler who always applies a false label, and the labelapplied by the labeler is disregarded or deleted. In addition, acountermeasure may be taken in which a weight coefficient to the rewardparameter Rw is extremely reduced. In contrast, when the reliability Rof the labeler is greater than or equal to the threshold value, thevalue of the score S(j) determined in the formula (1) becomes larger,and the labeler has a greater influence on the determined label and thecertainty degree calculated by the label determination unit 112 and thecertainty degree calculation unit 115.

The reliability R of the labeler is calculated from the labeled resultas described above, and thus the malicious labeler who applies a falselabel can be easily identified. Further, when the label is determined,the label applied by the labeler having the higher reliability isprioritized, and thus the accuracy of the label is improved. Inaddition, a labeler whose labeling accuracy occasionally changes withoutmalice may be alerted when the reliability R is reduced. Accordingly,the labeler can be notified of an educational benefit for correctlabeling and prevention of a careless mistake.

Next, a processing procedure of the information processing apparatus50000 according to the present exemplary embodiment is described withreference to FIG. 17. First, in step S5001, a target labeler performslabeling.

Next, in step S5002, it is determined whether labeling is completed ontarget learning data. A criterion of labeling completion is either orboth of that the certainty degree is greater than or equal to thethreshold value, and that the similarity degree between the determinedlabel and the output label is high. When the labeling is not completed(NO in step S5002), the processing proceeds to step S5011, and when thelabeling is completed (YES in step S5002), the processing proceeds tostep S5003.

Next, in step S5003, the labeler evaluation unit 119 calculates a labelevaluation value based on the similarity degree calculated by the labelcomparison unit 113.

Next, in step S5004, the labeler evaluation unit 119 calculates thereliability of the labeler based on the formula (6), and the label andattribute information management unit 110 updates the labelerreliability.

Next, in step S5005, it is determined whether the labeler reliability isgreater than or equal to the threshold value. When the labelerreliability is less than the threshold value (NO in step S5005), theprocessing proceeds to step S5012, and when the labeler reliability isgreater than or equal to the threshold value (YES in step S5005), theprocessing proceeds to step S5006.

Next, in step S5006, the reward parameter Rw is calculated based on theformula (5).

In step S5011, the data extraction unit 116 repeatedly extracts data andrequests the labeler to perform labeling until the labeling iscompleted. However, the data extraction unit 116 is not limited to amethod for extracting one piece of data at a time, and the dataextraction unit 116 may first extract data pieces as a whole andrequests the labeler to perform labeling until there is no more data.

In step S5012, when the labeler reliability is less than the thresholdvalue, the labeler is recognized as the malicious labeler, so thatinformation of the label applied by the labeler is deleted, a weightthereof is reduced, and the function f(R) in the formula (5) is set tozero or a very small value.

According to the present exemplary embodiment, the labeler is evaluatedbased on the labeled result, and the labeler reliability is updated, sothat a label which is more reliable and has less possibility of errorcan be obtained. In addition, the reward of the labeler is determinedbased on the evaluation of the labeler, so that improvement ofmotivation and work efficiency of the labeler can be expected. Further,the malicious labeler is identified, and thus harmful effects ondetermination of the label and generation of the prediction model can bereduced.

The configuration of the information processing apparatus according toeach of the above-described exemplary embodiments can be used inappropriate combination.

If a false label is applied to learning data, the label can beappropriately determined.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, the scope of the following claims are to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2016-249170, filed Dec. 22, 2016, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus whichprocesses learning data used in learning of a dictionary for performingpredetermined recognition, the information processing apparatuscomprising: an obtaining unit configured to obtain information of aplurality of labels applied to the learning data by a plurality ofusers, information regarding reliability of each applied label itself,and information regarding reliability of a user who applies the relevantlabel, wherein the information of the label is information regarding aresult to be recognized in a case where the predetermined recognition isperformed on the learning data; and a determination unit configured todetermine a label to the learning data from among the plurality oflabels based on the reliability of the label itself and the reliabilityof the user who applies the relevant label.
 2. The informationprocessing apparatus according to claim 1, further comprising anevaluation unit configured to derive an evaluation value for each of theplurality of labels applied to the learning data, wherein thedetermination unit determines a label to the learning data based on theevaluation value.
 3. The information processing apparatus according toclaim 1, wherein the determination unit determines a label to thelearning data further based on a likelihood of a result that aclassifier for identifying a label of learning data applies to thelearning data.
 4. The information processing apparatus according toclaim 1, further comprising a display control unit configured to displaythe plurality of labels applied to the learning data on a displayapparatus.
 5. The information processing apparatus according to claim 4,further comprising a label comparison unit configured to compare a firstlabel and a second label in the plurality of labels applied to thelearning data, wherein the display control unit displays a resultcompared by the label comparison unit on the display apparatus.
 6. Theinformation processing apparatus according to claim 5, wherein the labelcomparison unit calculates a similarity degree or a coincidence degreebetween the plurality of labels, and the display control unit displays aderived similarity degree or a derived coincidence degree.
 7. Theinformation processing apparatus according to claim 4, wherein thedisplay control unit has a function of displaying the label by filteringor sorting based on the reliability of the label itself.
 8. Theinformation processing apparatus according to claim 4, wherein thedisplay control unit has a function of displaying the label by filteringor sorting based on the reliability of the user who applies the label.9. The information processing apparatus according to claim 4, whereinthe display control unit displays another labeled learning data piecerelated to learning data which a user labels on the display apparatus.10. The information processing apparatus according to claim 1, furthercomprising a certainty degree derivation unit configured to calculate acertainty degree representing a likelihood of the determined label basedon the reliability of the label itself.
 11. The information processingapparatus according to claim 10, further comprising a learning unitconfigured to learn a prediction model based on the determined label;and a recognition unit configured to perform recognition using theprediction model.
 12. The information processing apparatus according toclaim 11, wherein the certainty degree derivation unit derives acertainty degree based on a result recognized by the recognition unit.13. The information processing apparatus according to claim 10, furthercomprising a data extraction unit configured to extract a data groupregarding which a request for labeling is made from among a learningdata group based on the certainty degree.
 14. The information processingapparatus according to claim 13, wherein the data extraction unitextracts data of which a certainty degree is low.
 15. The informationprocessing apparatus according to claim 14, wherein the data extractionunit extracts the data group so that the data group includes apredetermined number or more pieces of learning data which are alreadylabeled.
 16. The information processing apparatus according to claim 1,further comprising a label evaluation unit configured to evaluatelabeling applied to the learning data based on a calculated result of asimilarity degree between the label applied to the learning data and thelabel determined by the determination unit.
 17. The informationprocessing apparatus according to claim 16, wherein a label evaluationunit derives reliability of a person or an algorithm which performslabeling based on the label evaluated by the label evaluation unit. 18.A method of information processing for processing learning data used inlearning of a dictionary for performing predetermined recognition, themethod comprising: obtaining information of a plurality of labelsapplied to the learning data by a plurality of users, informationregarding reliability of each applied label itself, and informationregarding reliability of a user who applies the relevant label, whereinthe information of the label is information regarding a result to berecognized in a case where the predetermined recognition is performed onthe learning data; and determining a label to the learning data fromamong the plurality of labels based on the reliability of the labelitself and the reliability of the user who applies the relevant label.19. A storage medium storing a program for causing a computer to executeeach step in a method of information processing for processing learningdata used in learning of a dictionary for performing predeterminedrecognition, the method comprising: obtaining information of a pluralityof labels applied to the learning data by a plurality of users,information regarding reliability of each applied label itself, andinformation regarding reliability of a user who applies the relevantlabel, wherein the information of the label is information regarding aresult to be recognized in a case where the predetermined recognition isperformed on the learning data; and determining a label to the learningdata from among the plurality of labels based on the reliability of thelabel itself and the reliability of the user who applies the relevantlabel.